OOCOillT BBSOBE 



BD IBS' 695 



El 012 6tl0 



AUTHOR 
TITLE 



PUB DATE 
MOTE 



Drezek, S+an: Ai^d Others 
Current vs* Ideal Proxedures for 
Educational Proqran-E valuation Budgets; 
School Evaluator5?. ♦ 
Apr BO V ^ , 

U3f,: Pap«r preser.+ed at the Annual Mee 
American Educational Research Assoc 
Ht Aptil 7- 1 1/ 1980) . 



Deteininij 

A SupVey 



"fur^* 



of 




ing of the 
on (Boston, 



'EDRS PRICE 
DESCRIPTORS 



MP01/PC02 Plus Postaac. 

♦Adiinistrator Attitudes; *Budaeting; ♦Federal 
Programs; • Nat ional Surveys; * Program Evaluation; 
♦Resource Allocation; School Districts; Stat^ 
Departments of Education; Tables (Data) 

ABSTRACT \ ; . • 

; Bo^h smaller and larger School districts allocate 3 

percent of total budgets, or m,00 per pupil, fcr Eiogxam evaluation. 
Of t:his $i*,00, $3.00 is'scent on outcome or process evaluatioli, 60. ' 
cents on program planning/needs assessment, and UO cents on 
dissemination/utilization. Findings are from an evaluation budget 
procedures survey of evaluation directors of 55 local education 
agencies (lEA) and 1U state education agencies (SEA). A ma:^ri.ty 
re'comiehded a -^i-B percent, sliding scale, program cost allocation for 
-evaluatlpn. LEA respondents judged evaluation more important to 
subaequefiit program operation ttian to program funding. 
Misunderstanding, liigh costs, political considerations, and poor 
quality evaluations lower the priority that educational leaders hold 
for evaluation. It is reccmnendetT^ha t evaluators spend morcf time 
communicating and encouraging use of evaluation findings, (Author) 
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Evaluation Budget Procedures 
1 

' , Abstract 

. . ^ , ■ . ■. 

Both smaller (under 40,000 students) arid larger school districts 
►'aj^locate^,3% o£: total budgets for evaluation, or $4,00 per pupil ($2^00 locally 
'.funded). Of this -$4100, $3,00 is spent on outcome or process evaluatidn, 
$•60 on program-planning/needs-assessment, and $,40 6n disseiftdnation/utili-- 
zatlon; Findings are from an^evaluation-budget-^procedures* survey of "55 local- 
educatio^i- agency (LEA) and 14 stat'b-- education-agency (SEA) evaluation directors^ 
A majority^recorimiended a 4-8%/ sliding-?scale, program-cost-allocation for 
evaluation* 'At present program-cost-allocations for evaluation are 1-5% 
(,5-3% for some large programs). LEA respondents judged evaluation more 

4 

important to subsequent prograr.-operation than to program- funding. 
Misunderstanding, high costs, politic^il considerations, poor-quality . 

evaluations- -all lowtr the prio\^ty educational leaders hold for evaluation, 

* * 

Evaluators should allocate more effort to communicating, and encouraging 
use of, evaluation findings • 
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Current vs,'^ Ideal Procedures 



for /Determining Hducatiojial Program- Evaluation Budgets : • 
A Survey af^ School Evaluators / » 

. . .V - 

f\ . ■ / 

The average family probably , spends about as iiiuch per year ^6 assess the 
merit of its children's educatioh'al programs as it does to buy school lunches 
for the same children for one w6bk. This conclusion is based on, two recent 
surveys of evaluation act ivitie^' in U.S. public education. 

L/QU, Doscher, McGrariahan,.;and Williams (Note X) discovered that as^of 
1978 about half (320) of the ^79f6 U.S, public school districts enrolling 
10,000 or more students had departments, offices, or units responsible- for 
evaluation of educational programs. (The proportion of smaller districts * 
having evaluation units was eve^ less,) 

Among the 230 districts participating* in the- Lyoit et aK survey, the 
median 1977-78 per pupil expenditure for program evaluation, testing, and 
other research act^ivities of tjhre evaluation unit was between $3 and $4 
(with a range from $^10 to $90) • 

Average per pupil expenditures for evaluation in the largest school 
districts may be somewhat higher than the average for all school districts* 
The 35 respondents to a 1978 purvey by Webster and Stuff lebeam (Note 2) of 
the 60 largest U.S. school diistri^ts ihdglcatcd an average 1977-78 per pupil 
expenditure for evaluation, testing, and other research activities of the 

y . *' • 

evaluation unit of about $6.00 (SU about $4.00), with a median per pupil 
expenditure of about $3.00.* (Tho,«c averages may be slight underestimates 

/ 7'. 
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since in a small jninctrity of these districts the pilpil testing function was * 
carried on outside the evaluation unit*) 

.Both the Lyon et al. and Webster-Stufflebeam surveys suggest that during 
1977-78 U^.S. sbhool distric^ with evaluation units ^pent about one-thirdi 
of one percent of total district expenditures on the program evaluation, 
testing, and other research activities of those units • ^ 
' Concern about inadequate funding for evaluation of public-education 
programs--as well as concern about other aspects of educational evaluation-- 
prompted* the American Educational Research Asisociation's division on School 
Evaluation and Program Development ^Division H) to organize in 19/8" five 
task forces or committees. Each task force was. asked to prepare by !l980 
a pasition paper on a 'particular issue in evaluation. 

The present paper represents par.t of the work prerequisite to the 

position paper of the AERA Division H task force on resource allocation and 

budgeting for public School evaluation* This p^per reports results of a 

1979 survey of heads of evaluation units in public school districts (here-* 

• * 
after referred to as local education agencies, or LEAs) and state depart-- 

ments of education (state education agencies, or SEAs) . 

This survey primarily concerned current vs. ideal procedures fpr . 

determining program evaluation budgets in LEA^; and to a lesser exteftt, in 

SEAs. An additional purpose of this survey •was to assess perceived reasons 

« 

ifoXL^he generally acknowledged, low fiscal priority federal, state, and 
JLocaJ' educational leadc^rs hold for LEA evaluation actij^ities* 

The ^low priority accorded allocation of resources for LEA evaluation . 
belies the public commitment of many educationai administrators Wd school 
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^oardsl to. accountability; .that is, to assessing the types and degrees 

• of impact of educational programs and expenditures on student development. 
The nie'ager ^funding of evaluation activities- -and the fact that only half of 
major (J.S. school districts have evaluation units--suggests. that the verbitl 

* emphasis on LEA accountability pays only Up service to support of LEA • 

* • y 

evaluation, \^ 

Again, one purpose of the present ' survey is to obtain evalua.tor's , ^ 
opinions concerning reaisons for the poor financing of LEA evaluation and^ 
research. 

The low priority of LEA evaluation financing may be inherent in thO:' 
fact that evaluation \% a new j(but increasingly prevalent) activity, elbsely 

^ correlated with the increase in federal funding for public education/ A^out 
85% of the 320 LEA evaluation units in districts of 10,009 or more students 
we,re es,tablished during or after 1965 (Lyon et al,)* the year Congress 

"^signaled a massive increase in federal involvement in public education by . 
passing the Elementary -and Secondary Education Act (ESEAj) ; abou^ half of • 

thtjse evaluation units were organized after 1971* 

.< 

For many LEAs, the need to' provide evaluative data concerning student 
participants* in federe^lly funded LEA programs— as a condition of continued ♦ 
funding--may have been the main impetus for organizing evaluation unit • 
Local commitment* to use of these or other evaluative data has not irtways 
been necessary to establishing an LEA (Valuation unit. \ 
• Metropolitan school districts have the greatest eligibility for federal 
assistance. These districts al§o tend to be the ones with evaluation units. 
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Federal funding for public education largjoly emphasizes programs targeted at 
children of the urban poor (e.g., compensatory education, desegregation 
efforts, bilingual education^ etc.)*/ LEAs can, and must, use part of their 
federal funds for these programs to gather evaluation data. Hence, Lyon et 
al. reported that as of 1978 89% of metropolitan"1!&Cr (enrolling ^or 45,000 
students), 59% of "large" LEAs (25, 000-44,999), but only 33% of "medium"- 
sized LEAs (10,000-24,999) had evaluation units. 

1*erhaps local school administrators see such evaluation data as a 
federal bureaucratic requirement — a requirement to be met as efficiently 
and as inexpensively as possib4e. Perhaps federal, SEA, and LEA administrators 
haVe little incentive to apply such evaluation results at the local level to 

improvQ these federally funded programs. Perhaps, educational administrators ' 

« 

have inadequate experience or training in, using evaluation results as a basis* 
for funding or improving* programs • , • ' 

Finally, LEA-produced' evaluation efforts may have deficiencies in 
scope, design, or execution that limit their usefulness in educational 
planning* LEA evaluation units — like other LEA units and programs — must 
demons t,rate t^jeir value and j^^ccount ability. Any increased share of 
educational resources for evaluation .must be based both on a rationale fox 
such increased budget allocations and also on a reasonable expectation that 
such increases will lea<l tcf improved data for pirogram planning. 

Observers of LEA evaluation, both from within and outside the ranks of 
LEA ©valuators (e.g., AERA Division H task forces vs. the National- Institute- 

I - 

of-Educati^- funded study by Lyon et al.) aye now compldting the first 

. / ■ . • ■ 
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•^report cards'' for LEA Qvaluation unit activities/ 4 ^ 

* *» , . 

This paper s6eks to supplement these critical asse'ssments of LEA 

evaluation by providing «choorevaluators» reports of curreift procedures for 

determining evaluation budgets; their opinions. concerning ideal budget ♦ 

procedures; and their perceived rcas6ns why allocation of resources for LEa!\ 

program evaluation does i\ot have a higher priority. / . ' 

^ Method 

A questionnaire w.as mailed to 204 school evaluators invearly 1979. 
Subjects - 

LEA evaluators. A survey mailing list of 157 heads LEA evaluation 
units was compiled primarily from. the following two seurces (avoiding 
dupMcations) : (a)'4i^^ h^?ads of evaluation units in districts enrolling 
10,000 or more students,- representing about gne- third of the list compiled 
for the Lyon et al. study; and (b) 33 respondents to the Webster-Stuffle- 
beara survey o^ heads of evaluation units Vn the ^0 largest U.S. school 
.districts. 

The 55 LEA evaluators responding to the questionnaire represent 
return rate. Respondents were categorized as representing smaller L5As 
(average daily m«ml}ership of less than 40,000 students in 1977-78; 30 
respondents) or larger LEAs (40,000 or more students; n = 25 respondents). 

SEA evaluators . A list of 47 evaluation officers representing the- 
evaluation function in the educationaH agencies ^ most states and, several 
U.S. territories was obtained, from the Committee «fi Evaluation and Information 
Systems, formed by the Cbuncil of Chief State School Officers. < 
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The 14 SEA respondents represented a*25% return rate. 
The geographic representation of LEA and SEA respondents is ^ovm in 
Table 1. * ' 



Insert Table 1 about here 
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Questionnaire 

The survey questionnaire was prepared by one of the authors, with 
' assistance from members of the AERA Division H Task Force on Resource 
Allocation for Program Evaluation* 

Respondents were asked to answer factual items concerning^ LEA (or SEA) 
organization for evaluation; enrolment; total agency budget, and budgets 
for operation and evaluation of "specific programs; and then-current 
procedures (including any percentageyrof-program-costs guidelines) for 
determining^^fldgets for program evaluation. Respondeptjs were also asked to 
estimate thb relative costs of each phase of a "typical" program evaluation. 

Additional items asked for descriptions of factors that would make a 
program evaluation coli^lll^e than, or less than, the normal range of 
percentages of program costs. Assessjigats ^6f the inflXience of L^A (or 
SEA) evaluation reports on subsequent prog/am operation or funding were also 
requested* Respondents were asJced to suggest reasons why allocation of 
resources for program, evaluation does not have a higher priority among 
federal, ^ state, and local educational leaders. ^ 

•nr — ■ ' — 

Finally, respondents were asked t6 select one of several suggested 
^ procedures for det^mining the 'total budget, of a particulat program 

evaluation. * . 
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Results 

' The first results presented describe the level of support for LEA 
evaluation, in relation to' LEA enrollment and budget. Succeeding Results 
sections describe types of evaluation^-unit organization, methods used to 
determine. overall budgets for program evaluations, levels of funding for 
evaluation of federally funded LEA programs, percentages of evaluation 
funds spent on each phase of a typical evaluation, reasons for extra- 

i 

ordinarily high or low perce-ntages of program costs being allocated for 
evaluation, respondents^ judgments concerning the influence of their units • 
evaluation reports, and reasons why program evaluation does not have a 
highejr priority among educational leaders.^ Presented last are respondents V 
recommendations concerning ideal procedures, and, ideal overall funding 
levels, for program evaluation. 

Enrollments, Budgets, and Overall Level of Support for LEA Evaluation 



Insert Table 2 about here 
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Table 2 shows the range of enrollments among the LEAs represented in 
this survey was 1,000 to 600,000. The LEA sanqple included eight districts 
with enrollments over 100,000.. The median enrollment for smaller LEAs was 
about 18,000; for larger LEAs, about 75,000. Taken together, these LEAs 

* 

enrolled a tota^ of 3,617,000, students, or nearly 7% of the 1977-78 U.S. , 
school-age pojjulation. 

The range of total evaluation budgets for the surveyed districts 
was also large— from less than $100,000 to two-thirds/of $1 billion for 
total LEA .budgets; and from the equivalent of 1 salaried teacher ($16,000) 
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to over $3 mil'lion for LEA evaluation budgets. ^ 
Indices of average support for evaluation in smaller vs. larger LEAs 
* were calculated using median evaluation budget figures shovm in Table 2. The 
discrepancies between the mean and jpedian for the two evaluation budget 
statistics--total 1978-79 LEA evaluation budget, and local funds allocated in 
ia78-79 to evaluation--reflect the fact that a minority of both the smaller 
and larger LEAs give the evaluation unit functions in addition to program 
evaluation. Table 3 suggests these extra functiqns can include dia^riCjt- 
wide pupil testing, management information systems, or planning and grants- 
manship for new educational programs. Because these extra functions may 
require substantial funds beyond the basib budget for progra^ evaluation, 
the median evaluation budget figures were judged more representative tlian the 
•mean to' show average amounts allocated for evaluation in smaller vs. larger 
school districts* , 



Insert Table 3 about here 



Both smaller and larger LEAs allocated, then, on the average, the same, 
.3% of their total budget for evaluation; » or slightly over $4.00 of a mean 
per pupil expenditure of $1,500 for the smaller LEAs and $l,7ob for the 
larger LEAs. ^ . 

For both smaller and larger LEAs, about half of the total evaluatioij^ 
budget came from local funds, on the 'average. 

In sh^rt, in both smaller and larger LEAs, about $2*00 pef pupil was 
allocated for evaluation from local fiifids, and another $2.00 per pupil ^was 
80 allocatjod frofi other sources (presumably mostly federal, but tflso some 
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state/ funds). 
■ Organization of the Evaluation Unit 



Insert Tables 4 and 5 about here , ' " - 

' ~~ ' * ■ • . 

Tables 4 and 5 suggest. that the njost common patterti of organiz^tiort for 
evaluation in both smaller and larger LEAs is an evaluation unit headed .by.' 
a "Director of Evaluation" (or highly similar title), who in tUrn reports t-o * 
a supervisor just below the levgl of superintendent; .e.g:,. an assistant, 
associate, or deputy superintendeik. Aboi^t^ one- third to one-fourth of the ' * 
heads of. evaluation units report directly to th^ LEA superintendent,. Tables 
4 and 5, however, show that a variety of other organizational forms are used. 

• Air.ong SEAs rep^oseuted in the survey, tHe modal form of organizati'on for 
evaluation is a unit headed by a U»Lrector pf Evaluation, who reports to an 
SEA Deputy Commissioner.'^' *• 

Methods Used to Determine Proj»ram- Evaluation Budgets , ■ 

In about half or more of each surveyed group— smaller LEAs, larger LEAs„ 
and SEAs--the total ahiount budgeted for evaluation \of a program is determined 
by the scope'of evaluation work written jinto the prijgram (see Table^ 1). 



Insert Table 6^ about here^i 

■■ : — r 



Wlien evaluation budget amount is tied t6'the scope of evaluation work 
described in the program proposal, the range of percentages pf program costs 
represented by the evaluation budget is bounded by a median low of 1.5% to 
a median- high of about 5.5%, . 

^ Four about ono-fifth to one-fourth of the LEAs, a roughly, fixed perccttt- 
ago (usually about 5%) of program cost:s is allocated for program evaluation. 
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V. • one-fourth* pf the smaller LO^s, but in apparently very 'few larger 
LHAs, the evaluation unit ^eeks as^much money as pfcis€4;bl:fe for evalyation of. 



a pragratii, sinc^e--ihe unit s^^idonr received suff ici6«t\fundiT\g 'tp* conduct 
• evaluations.^ In this* sitwatioh of iuadequaj^e funding,, the ujual perQentag^s 
. of 'f)rpgrain bests allocated fPr evalu^t^on ^fa^g^ from a^ median lowj^f* about 

1^ to a Median higJ^ of about ^ * ' ■ • :r J / . " . 

■ Apparently, about one- third of; the LEA uaits use ^ combination of irtethqds 



IP 
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Cc^gti chec|.eJ mbre th^jtn one method/ wrote "all thiree" described here-, . • 
*'no fixed rule, '^/etc. ) > or ^se other methods (e,gA -consider, salary lovets 
-c4' staff available to work on tlie evalaatiou). in •Sptermining a budget for a 
•V paj-ticular evaluatio.i). * • * % * ' 

' r ^liy^luation Budgets for Fe d erally F-iindei^ -[.LA Programs ^ ^ ; 

^ Allbcation,^^ from fed'^ally funded UEA programs provide staple^and 

\^^f^.i * ... - - ' ^ - ' 

V^^V^i^i;^* reliatiyely stable, bread-and-butter sustenance; for many %EA evaluation janitfis; 

* pa^tI'cuJ5^arly^;in urban LEAs, which are often eligible fot considerable federal 
funding. *^ • ' / /^J if! 

■■- ^ ^ ■ ' .%s^!^^ table ^ ;7Sftfe lier6 - . 

^ , ... ':^?^ 4W " ^ 

" -.Ttlble. 7 suggests that the most imjport*an|;|ijD5derally funded LEA program— 
• • in terms 6f the number of LRAs having such a prbgtam, and in terms <)f median 
V atimtatl. by^ijgeiir.-is ESEA, title I, prpvi^ling compensatory, education for 

^ edacationaUy dt^cjfantaged children in loy-inoome ayeas. Thre.e- fourths of 

• .- ' .-.••iv.^ > ■ '■■•'■'""■*.,■ _ . ' , . 

. , tU« 55"LEAs,;:hjia 'suc^^^ |Th^ inodi^^ annual budget of $2;46o,bdo 

. ^'Kp^^fu^(iedtijHA .:i'iti6 I program^was nearly twice ^that foi* funded limergency 
-X^^^ (found; in only onO|third of the 

•. .- •■■<•* .««*••:• •.• •■ •! .... . ;■ .■;.■**"■!" / ^ 

'•. ' . 1' •;,••••,',•'••'•'»' .. •" •.. . ' 'v ' ■.•.■"■••■I I f 



. ' ' Evaluation Budget Proco4ui*es 

• . » . 

LliAsJ . In districts having Title I progriiins,<'the median annual evaluation 

budget for -this progrSin was $35,00Ci. . . 

* • 

For funde# Title I programs, the median percentage of program costs 
^allocated to the evaluation unit for Title I evaluation was about 1.5%. (For 
each LEA with^a funded Title I program, the Title I evaluation funds made 

available to the evaluation unit were divided by the total Titles 1 program 

•'■ • ■ .■ *■ 

budget. The median of these. tKA statlsticSr* rounded to the nearest half 

percent, is the median percentage' of program costs allocated to the evaluation 

unit.) ■ 

■ * ■ ' . ■ ^' 

One. LEA with a Title I program "provided no federal funds for Title I 

evaluation to the evaluation unit. Similarly, Table 7 showa that fgr each 

fe^ral prograp, at leasr one LLA had the program but allocated no f^eral 

■funds' to t^! evalu<it;on unit to evaluate the program. In such ca§es>1il)Hever, 

evaluation activities may have been conducted outside the evaluatioJ 

Other common federally funded programs- -found in about half of bol 

smaller and larger LHAs— were ESEA, ;^Ule^V-C,, ihnpv^tive^curriQulum 

development pr'bjects; and P.L. 94-142 special education programs. 

•v ; Title^lV|:C;;;I)'.rograms were generally the least expensive federal programs, 

with iiw^liajrf- annual buUgetjs for funded programs of .$50,000 for smaller LEAs 

and $250,000 for larger LEAs. » 

.% ■ „ . ■ . 

/ Ironically," the$o least expensive federal programs were associated with 
the highest me<li]^ percentage of prograjn cost*. ail6catbd to the evaluation 
unit for evaluation, 5%. The most expensive program. Title I componsatory 
e^uca,tlon, provided one of thei^west median -percental allocations for N 

evaluation, lk5%. The only lower lovol- of effort for support of WiA federal 

. ■ • • . ■ ■■ ■ " V , 
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program evaluation was that pro'vided by special education programs; namely, ' 

c, ■ 

.5%. ' 

Percentages of livaluation FUnUs Spent During Each Phase of a Typical Evaluation 
Respondents were asked to estimate, for a typical program evaluation, the , 
percentages of allocated evaluation monies spent o'n each of the following four 

phases: (a) program planning and needs i^assessment; (b) the gathering of process 

• _ , <' 

evaluation or implementation Valuation data; (c) the gathering of product 



QT outcome data; andy(d)» the dissemination and use of the data* 
^ Insert Table 8 about here 



Of every "$1(7 allocated for evaluation of a,, typicaTprogram^ slightly 

- * • . . - • 

over $4 were spent' on product or outcome evaluation activities; $2^ on process 
or implementation data gathering; nearly $2, dn program planning and needs 
assessment; and $1, on dissemihation and utilization (see Table 8)^* 

These average proportions of funds spent during each evaluation phase 
were similar for both larger and smaller LEAs. 

Represented SliAs apparently spent proportionately less evaluation money, 
on the avetage, than LEAs during the product/outcome evaluation phase^ and 
slightly more during the program planning/needs. assessment phase* 

The variability among LEAs in the proportions of evaluation fiinds spent 

on each phase was hlgh» particularly for the product/outcome evaluation attd 
process/implementation evaluation phases. About half the LEAs reported 

spending 30-50% of evaluation funds during the product/putepind: fcVft^ 

phase; and about half spent 20'-4Q% during process/:^plementft1tion evaluation, 

Only 6% *of the LFiAs reported spending more than 20li of. evaluation Ajnds in 
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dissemination and use of data. " , ' 

- ■ •■• \i ■ ■ ■ 

Factors Leading to a Higher or l^ower Than Average Proportion of Projyam 
Resources Being Allocated for Uyaluation * * ^ 

The 69 LliA and SUA evaluation directors were asked for written responses 
to two questions: ''What are some pf the factors which would make a program 
evaluation post more than the noj^al range and ''What are Some of the ^ 
factors wliich would make (^anj -evalpation cost "Isgss. . .?^' 

Content analysis of\ these open*:ended responses resulted in five 
categories of such characteristics^, or factors occurring during the conduct 
of an evaluation: measurement and instriunentation f£<^tors; other evaluation 
factors or reqmiroments; personnel factors; other evaluation budget factors; 
and a miscellaneous category of all other\ factors and comments. 

Associated with each factor was the potcentage of 69 respondents, 
nominating this factor as leading ^o either a higher than avcrtf^e, of lower , 
than average, prl^ortion of . program resources being allocated for evaluation. 

Presentation of these content-analysis results will be limited to^ 
factors nominated by at , least 5% of respondents as leading to high^ than 
average' evaluation costs; or nominated by at least 5% as leading to lower 
than avorage costs. 

Fjiictors leading to a higher than average proportion of program resources 
al located for evaluation . Measurement and instrumentation :^actors leading 
to higher than average costs were field callection of' interview and obser- 
vational data, .in addition to collection of paper-and-pencil data (22% 

■ V . . ■ ' ' ' ■ 4..., 'i . 

of respondents); developing new Or original tests oy quosti0Miaires (20%); 
purchasing commercially available tests or questionnaires (p%); and a large 
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noinbor of different ins^trunients and measures required by the evaluation 
design (7%) • * • ^ * * 

Other evaluation features or requirements leading to higher than average 
costs were largo program size or complexity (i.4%) complex evaluation 
dcsi£n--e,g. , control groups, evaluation audit, (^tc.-«(10'0; and, heavy data 
^aneilysis and reporting requiferftents (10%) • 

Personnel factors leading to higher costs were aynilability of well 
qualified staff, commanding higher salaries (12%); or private consultants or 
outside research firm conduct' the evaluation (10%). 

Under miscellaneous factors, data processing cost5--e,g», custom , - 

programming, time, services, etc. --were nominated by 17^ of respondents as 

/ 

a factor leading to higher than ^average evaluation costs. 

Factors leading to a loyrer than average proportion of program resources' 

allocated for evaluation. ■ Measurement and instrumentation factors leading 

to more economical evaluation included using already available data, and/or 

already puiail^ascd or developied tests aftd questionnaires (16%) ; evaluators 

and program staff share the wojrk or cost of dat^ collection— including iise 

■ . ' ' ' * -■ 

of diagnostic tests usable JFor both program operation and evaluation (7%); 

and a small number of different instruments antf measures Tequired (6%). - 

Other evaluation features or requirem^ts leading to lower costs were 

>■ . 

light data analysis and reporting requirements (10%); small program size or' 
♦complexity (6%); and a design that is/elegantly simple, or evaluation 

proceiduros that tr\. familiar or well standardized (6%). 
-t A totar of 13% 0^ respondents did not write a response tx) either item 

about cost factors. . * 



/ .■ ■ ; 
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Respoml^ts^. Judgments Concerning the Influence- of Their Units ^ Evaluation 



Rep o!rt$ «^ ' 

r ' ' / • * • . . 

LtA e»al,UAtion,;^eports are apparently more important in determining the 
'way programs are operated in subsequent years than in determining overall 
progrdm fiinding in subsequent . years • % ' ' 



Ins ert.' Table 9 about here 



/ About three- fourths (78%) pf the 55 LEA' evaluation directors judged 

• *. . ' • , . 

/ ' • * 1 

■ ' ' ' • t 

their uni^^s' evaluation report:^ to be an important factor in subsequent 
prpgraijv ojieration. About half (45%) judged these reports important in sub-- 
sequent dctbrpinations concerning overall program funding (see Table 9). 

' Only about a third of the SEA evaluation directors judged their •units • 
Walu?Vtion reports as important to either funding or program operation 



decisions. 



Reason:^ Allocating Resources for Program Evaluation Does Not Have Higher 
Priol^ity' Among Educationa:! Leaders 

The LEA and SEA evaljiator directors were asked, in three questions, to 
write their perceived reasons why educational leaders do not a#sign a higher 
priority to evaluation. The first question dsked respondents to "^tate the 
two Tnost important reasons why you believe resources for program evaluation 
might hot fiaVe a higher.priority with federal funding agencies." The highly 
similar second and third questions ask.ed ffit reasons why such resources might 
iftot have, a higheir priority with "SHA leadership," or "LliA leadership/ 

*wpectiVo4y. '^^Vr.^. ' . ' "y 

^ Content analysis of these written responses led to four najot categories 
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of iinportarjt reasons why evaluation doe's not "have a higher priority; high ' 
cost, 'budget prpblems, or competing priorities; usefulness' and "relevance 
of ev^iluation; misunderstanding of ejfaluation- methods or purposes- -fear, or 
ignorance; and quality control and staffing- problems.^ ^ * 

Presentation of this cbntent analysis will be limited to those realons 
nominated by at least 5%^ ot respondents as accounting fojr evaluation nQt. having 

a higher priority amohg one or more of the following groLps: LEA leaders, 

'' . ' f ' " 

SEA leaders, or federal funding agencies. » ! ' 

High cost; budget problems, or com^^eting priorities' . The too costly 

nature, of program evaluation (e.g., "tight budgets," "limited revenue," 

etcO was offered by 28% of the 69 LEA and SM .evi^luation diDifcctors as an 

important reason for evaluation not . having a higher priority among LliA 

leadersldp; 17% of respondents nominated this as a reason for evaluation 

not having a highei% priority aniong SEA leadership; ar\d 19%, as a reason 

V 

for evaluation not having a higher priority among federal funding agencies • 

Other cost reasons for evaluation not haying, a higher priority were 
less frequently nominated • The fact that direct service to students has a 
higher priority than evaluation wds judged an important consideration . for 
LBA leaders (13% of respondents) and federal funding agencies (?%)• Lack of 
legislative action supporting evaluation was judged an 'important consideration 
for SEA loaders (9%). Also relevant for SEA loaders was ^EA emphasis on * 
inonitoring programs' compliance with regulations rather than evaluating program 
effectiveness (6%).. ' ^ 

Other ijjpprtant reasons sugg^ostod wore that LEA leaders do not Kavo * 
enough time to study and use evaluation result^ (6%); and that federal agencies 



. ) 
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have.otho-f servdces and pressures with higher priorities than evaluation (6%). . 

Lack -of understanding or misunderstanding of evaluation methods olr 
purposes; and defensiveness . Educational leaders often don't understand the 
methods or purposes of evaluation, or aren't accustomed to using evaluation 
results for decision-making. Suchstatements were offered by about 20-30% 
of respondents as important reasons preventing a higher priority for evaluation 
at each level-- SEA, and federal. Individual respondents suggest e4/tij(A • 
educational leaders '.'don't know how to use results when they get them," 
have ''unrealistic expectations/* are ^'not trained or qualified to use or 
interpret results," or "donH perceive the value of evalua^tion %xi overall 
program development." • ^ 

Educational leaders see evaluation as a threat to their jobs or . 
authority, particularly at the LEA dnd SEA levels (according to 12%, and 7% 
of respondents, respectively). ' ^ • 

Federal funding agencies may underestimate the actual use made of 

evaluation results at the LEA level (7% of .respondent?) . 

Usefulness and relevancy of evaluation ^ Proportions of 13-20% of 
^ * • '•^ 

respondents suggested that evaluation results are frequently not 5ieen as 

useful, or are not used, for improving or funding .programs at the LEA, SEA, 

and federal levels. This lack of usefulness, or use, limits the priority 

given to evaluation. 

Reasons for this lack of use, or usefulness, offered by 7-14% "of 

. . rospQndfhts, include the inconclusivcriess of results se.nt to federal agencies; . 

the, federal and LliA planning and funding of programs for primarily "political 



:eUure^ 
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reasons; and the carrying out of SliA and LEA evaluations Independently fr-om 
program planning and development. 

t ^ 

Quality control and staffing problems . The poor quality of some 
evaluation research may be important in evaluation* s lower priori ty>aja©T?g ^ 
SKA leaders (aTccording to 10% of respondents) . 

Problems in finding qualified^ competent evaluators may prevent a higher 
priority for evaluation a^^-the LEA level (according to 14% of resppn^ents) . 

t , ■ V ■ 

Evaluation Directors* Policy Recommendations Concerning Allocation of Program 
Resources to Evaluation ' - • 



Insert Table 10 about here 



A majority of each respondent grouji — evaluation directors in smaller 
LEAs, larger LEAs, and SEAs — endorse a sliding scale approach for the 
all'Dcation of program resources to evaluation (see /Table 10). 

Respondents 'were asJced to choose among thfee alternat^ive policies 

■ ■ ■ ■ - ■ ■ . \ 

that the Task Force on Resource Allocation might support. Proporn^ns of 
52-59% of each group recommended adoption of '^a sliding scile with lower 
percentages^s program costs exceed certain levels." 

• The other two policies — fixed percentage of program costs for 
program evaluation'' and "a minimum, percentage of progx^am costs for program 
evaluationJ*--were each endorsed by nearly 20% of the total LEA respondents. 

The fixed percentage policy had more support "among the smaller LEAs~* • 
(28%) than among the larger" LEAs (9%). .. 

. ' The minimum-percentage, approach had nearly equal' support among smaller 
and larger LLAs, (17% ^ind 18%, respectively); 15% of Che SEA evaluation 
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latipn 

... 

uirectorjs also supported the minimtun-percentage approach, 

A number respondejit$--10% of the^ smaller LEA evaluation directors, 

♦ ^ 
18% of the larger LFA directors, and 31% of the SEA directors--did not 

/ f 
recoiiunend Task Force support for one of the three positions here described* * 

Instead, they recommended* some combination among these positions, ofr^lse 

specified yet another alternative ppsition. 

Evaluation Direcctors^ Recommended Percentages of Program Costs To Be 
Budgeted for Evaluation . / . ^ 

r ' : _X 



Insert Table 11*^ about here 



Evaluation directors were askad to answer tlie question, •^What percentage 
of program costs should be budgeted for program evaluation?*' by provitling e 
percentage range. The obtained range extended from 0% to 20%. For LEA 
evaluation directors, the average value given for the lower bound of the 
range was about 4% (SD = 3%)-; the. average valufe for the upper bound of the 
rahge, about. 8% (SD = 5%) . 

Evaluation directors representing smaller LEAs provided slightly' Kigher 
mean percentag^fes, for this range than did evaluation directors representing 
larger LEAs (4*5% to 8.1% vs. 3.0% to 7.3%, respectively). 

For SEA evaluation directors, the mean lower bound of the recommended ^ 
range was aboi^t 5%; the mean upper bound, about 9%. \ .. . ' 

Discu$sipn / 

• The purpose of thii^ paper has been to survey school evalua/ots concerning 
^current procedures for determining evaluation budgets; thei-x opinions concern-^ 
ing ideal evaluation budget amounts and budget alloqation pr</cedures; and 
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their perceived, reasons why allocation of resources for |.EA program 
evaluation (as well as SEA" and federal educational evaluation) does not have 
a higher priority. , 

•The statement that LEA program evaluation does not have a high enough 
priority does, of course, beg the question of what is the appropriate 
priority for surh evaluation activities. - v o j. 

This surVey suggests that LEAs Vdth evaluation units n6jf.^^budget about ^ 
$4,00 per pupil per year C$2.00 of this from local funds) for those units' 
activitics--such as program evaluation, , pupil testing, and other research. 
These resu^ are consistent with average LEA per pupil evaluation expendi- 
tures of $3-4 annually found by Lyon et al. and $5-6' found by V/ebster.;and 
Stuff leboam. The present survey also duplicated these investigators 
finding that LEAs allocate about ^3% of their totc^ district budgets ^for 
eva Illation, 

For educational programs with an evaluation component, the present 
survey suggests that 1-5% of the total bg4gets of these" programs are^ 
ol-dinarily allocated for evaluation. For some .of the largest- federally 
funded proiraras (e.g... Title I and special education) these evaluation 
allocationsl average .5-3%. For federally funded (e.g.. Title IV-C) and 
nonfederal programs emphasizing innovative approaches to education- -where 
dojaim^tation of program acceptance and effectiveness may be more' strongly 
. emphasized-- the percentages of program costs aiiocated for evaluation seem 

to average 4-5%. " . . " 

.Returning to the question of what is an appropriate priprity for 

# «- . * 

educational prcl|fam evaluation- -It seems unlikely that the percentage of 
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printer's ink allocated in the educational pres^«^to educational leaders' 
concern fox accountability and program effectiveness is consistent with an 
average LEA allocation for evaluation of .3% of the total LEA budget. 

The results of the present survey, of course, do not resolve the 
question of what fiscal priority evaluation activities should occupy* Further 
research specifijbally targeted at this question is needed, ^uch studies 
should address the current and potential quaUty, usefulness^, and benefits 
vs» CQstJ of evaluation. ' ' - . 

Evaluation unit activities could come to have a higher priority among 
all LEA activities if LEAs more frequently adopt an ompirical,^ research-- | 
based approach to all aspects of school administration-^heyond tlhe use of* 
evaluation results for management of iudi^vidual, experimental instructional 
programs. ^ 

For exampiy,, the evaluation ^nit might become .a useful resource for 
LEAs wishing to develop more rational bases for personnel selection and 
promotion; for comparing the cost-effectiveness of alternative methods apd 
materials for instruction; or even for planning school closings, school 
energy- conservation efforts, or changes In the school-lunch progtam. , 
*The primary concern of the present survey, however, is with the priority^ 
of evaluation within particular educational programs, more than with the 
priority of evaluatioji within LEAs generally* 

The present survey shows that a nationwide Samj^e of heads of LEA 
evaluation itiits recctomends a range of percentages of program costs for 
evaluation- -from 4-8%, on the average— that exceeds the currently aA^lable 
average percentage-of-program-costs allocations for evaluation. 



■ ^ Evaluation Budget Procedures 

* « -.23 V 

A. majority of these LEA evaluation directors (as well as a majority fef 
the SliA evaluation directors represented) rdcoiiunends a sliding, scale approach 

* * 

to determining allocations of program costs for evaluation, with lower 
percentages as program costs exceed certain levels • A majority of the 
evaluation units represented currently use such a sliding scale approach, 
wiiK the evaluation budget determined by the scope of evaluation, work 
requdred.- The current average range of percentages of prpgram costs for 
evaluation when such sliding scales are used is only about l*5-5,5%. 

The surveyed evaluation^^directors gave mixed reviews of the influence 

* ■ > 

their own units * program evaluation reports had oiv^ program^ operation and. 
program funding in subsequent years. Apparently, LEA evaluation reports are 

s 

more influential than SLA reports; and LEA reports are more important in 
determining tlie way programs are operated in subsequent years than in - 
determining overall program funding in subsequent years. 

Whether or not LEA evaluation reports were perceived as important 
influences on program funding' or operation was not highly correlated, with 
district size. (LEA enrollment). (See Table 9, Also, appropriate point- 
biserial correlations between tliese variables were near zero.) 

However, there w6re modest point-biserial correlations between pef^entage 
tff total LEA budget allocated for evaluation and (a) influence of evaluation 
reports on subsequent funding (r = .21, £ = .07, 1-tailed) arid (b) 
influence of evaluation reports on subsequent program operation (r - .24; 
p »' .04, l-tailod). Thus there is ^some reason to believe that LEA level 
of effort for evaluation is related to ^le usefulness of evaluation activities 
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in decisions concerning progr^un funding and program revision. * 

Overall, about three- fourths of the LEA evaluation directors judged 
their units' evaluation reports to be an important factor in subsequent 
program operation; and about half judged;, these reports important to subse- 
qtient decisions concerning program funding. • . 

Comments of these evaluators stiggest p6Ssible reasons why evaluation 

■ • ' " ■ . ' ■ ■ * V ■■ ' ■ ■ ■ ■ 

does not have a higher importance in program funding and operation; or, ' 
more generally, why evaluation does not have a higher priority amon^g LEA, 
SEA, and federal educational leadership. According to these respondents, 
program evaluation is often seen as too costly; educationji^ leaders have 
little training or experience in using evaluatioii results; political '» 
factors "often intrude into program funding and oper^tidtJ decisions j and .. 
some LEAs may not have enough qualified staff to conduct high quality 
evaluation., * • ' 

Program evaluation is certainly a costly enj:erprise--particularly when! 
it involves direct observation progi;|m participants; purchase or construct- 
ion of special tests or questionnaires; and employment of trained^ experienced 
evaluators. 

Until evaluation leader^ have more consistent experience, incentives, 
or traiiiing to use evaluation results for management decisions, there, will" 
probably continue to" be high variability in the budget levels; and budgeting 
procedure used, for progrflm evaluation. If ctirr^nt conditions persist, 
perctntages of prpgram costs allocated for evaluation will often bo lower 
than the allocations evaluators yced to do adequate evaluation. 
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.---V- *• HducatK)ual wya'luators may wish to con*liaer the dosirability of 

jia-lIOpatin^ increased effort t^^^^^^^^^^^^ results: and 

encouraging educational leaders to use thes« results, the present Wvey 

: r:5.u'ggests that of every $10 allocated for evaluation of a typical program, 

about $7 are spent on collection and analysis of data>^but only abtilit $1 . 

, ... .. ^'V,,^^?'^^^ efforts to disseminate research results and encourage their use. ' 

' ;• In short, evaluators jnay neftd to assume jnore .leiider^liip in convincing 

■■'.-* ^ ■ ' , ' * ' ' 

' other educators 'of the use'Fulness of evaluation results.. Improved' - I • 

' : ' I ■ - • ' ' • ■■ 

commiyii cat ions skills aR\ong evaluators may be • nec,essary to improved 



allocation of program resources for evaluation--and to an improved pribrit;y 
of tivaluHtion un'it actiArties/within the LHa'. ' . ■ . ' 



•'*■■' • 
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- "Footnotes 
This paper w^^S^ originally prepare<J for discussion and use by the 

, American Educational Uosdarch 'Association Division H (School Evaluation and 
Program Uevelopment) Task Force on Resource Allocation for Program 
Uvaluation. This Task Force will complete a fiftal report and position paper 

* during 1980. The three authors each assumed an <equal portion of the work 
required for the present paper; hence, the listed order &f, authors is 
arbitrary." The authors wish to thank Freda llblloy. Chair, and other members 

'of the Task Force for useful comments ^nd support during conduct of this 
research. Requests for reprints should be^ mailed to Paul Higgins, University 
of Minnesota, Area Health Education Center, 2929 University Avenue S.E., 
Suite 344, Minneapolis^, Minnesota^ 5^5414. 



^Available, from Paul Higgins at the address given for reprint requests! 
^A table showing tlVo' compjletii^ijptitejit Analysis is available as per 

Footnote 1., .. .. . .. . 

^Complete content anaiyisiS =^vail^j>le as per Footnote 1. 
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fJeoiiraphic Distribution of Survey Respondents 



30 Smaller 

th an 40,000 stu d ents ^) 



.LHA evaluation directors 

•representing-- * 
LLiAs (lesj 25 larger *LL:As' (40,000 



or more studepts,^) 



*'SHA" evaluation directors, 
representing educational 
agencies in .13 states and 
1 U,S. territory 




Phoenix, Arizona 
Corona, Cal ifornia 
Fulsimi, California 
Irvine, California 
Venturtl, California 
Hartford, Connecticut . 
IVlieaton,^ Illinois 
Cedar Rapids, Iowa 
ChaliiVette, Louis ^uia ' • 
Frederick, Maryland 
Brockton, Massachusetts 
Ann Arbor, Michfgan 
Benton iiarbor, Michigan 
. Niles, Michigan 
Watorford, Michigan 
Fcirguson, Missouri 
St. Louis, Missouri ^ 
LincbLn, Nebraska 
Wbod Ridge, Xew J<jrsey 
Dayton, ^Ohio 

OKhihoma City (Putnanv City) , 

Oklahoma 
Beavferton , Oregon 
liugene, Orego^y^ 
Troutdalc, Oregon 
V Providence, Rhode. Island 
Corpus Christ i, Texas 
Pasadena, Texas 

Ogdeju Utah 
Yak/nrn, Washington 
1 iiMitional district of » 
1/000 students, location 
unspecified 



Fresno, California 
Los Angeles, California 
Lakewood , Co l(Orado 
Wilmington, Delaware 
Jacksonville, Florida 
Atlanta,\Qeorgia ^ 
Honolulu, Ha^ii 
Chicago, Illinois 
Wichita, Kansas 
Loi^i s vi lie, Kentucky 
A'ejv Orleans, Louisiana 
Annapolis^ Maryland 
Boston,*4Mass^<!liusetts 
Kansas City, Missouri' 
^Albuquerque, New Mexico 
Akron, Oiiio 
Columbus, Ohio ^ ' 
Oklahoma City (District 89) 

Oklahoma 
Philadelphia, Pennsylvania 
Memphis, Tennessee 
Nashville, Tennessee 
Austin, Texas 
Dallas*, Texas , 
El Paso, Texas 
Fort Worth, Texas 



Alaska 

Delaware 

Kentucky 

Maryland 

Mississippi 

New Jersey 

New York 

New Hampshire 

North Carolina 

South Dakota ' 

Texas 

Virginia 

Wisconsin 

Puerto Rico 



Note. The^ terms LFA arid SF^A represent local education agency (d.e*, public school 
district) and state education agency (i^, e. , department of educaJ:ion or public 
instruction); rcspoctively,. 6 - , 

Some plado hames (ovg«:v St*. Louis) repi^esent smaller LliAs serving 

only one part of the? named ma joi: city. OthejfV place names (e.g. , Lakewood, Colorado; 
Honolulu) indicate the center of a larger (regional) LliA extending beyond the named 
municlpaiity. / ^ / 

/B«»oU ill ^oApohdents' solf-reportod estimate of 1977^78 aVerago daily 



membtrsht^ 



7 



Table 2 

^ Solocted Chtractorlstics of UAs 



Chariict<?ri5tlc 



Approxlnnte 1977-^8 avorac« daily pupil 
cieiRbershlp, In tliousands • ' ^. 




Snmllor UAs 
(tt-30) 

« su 



Mdn 



(Df25) 
' SO 



Mdn 



18.3 



122.7 



9.1 



18.2 



154.6 



74.8 



M ' 



Total LEAs 
(a-55) 

SD 



65.8 



104«2 



Mdn 



56.0 



ERLC 



Number of years (including 1978-79) pro|raM 
•valuation effort has been orgonliiod in ^A 



Total 1978-79 budget of LBA (including 
local, state, Federal funds), in 
thousands 



Total 1978-79 LBA evaluation budget* 
(Including local, syte, federal funds), hi 
thousands 



UcAl fvmds Allocated in 1978*79 to 
evaluation, in thousandi 



[Jote. H • wean, SD • gtundard deviation, M^i • loedian, 
^n - 29. 



'n - 54. 



5.9 ^rfi„ 



$29 » 590^ 



10^0 yrs. ' 
5.6 yrs. 7.8 yrs. 

5.7 yrs. 9.5 yrs. 



$128^ 



l25,4«7^ 



$1W^ 



$27,996^ 



|165,p2S 



$685 



' $127,561 



$887 



$124,167 



7.7 yrs. 



$91,5|S5^ 



6.2 



yrs. 



6.3 



$«6* 



$140,518 



$674^ 



31 



$45,509^ 
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Tabic 3 



V. 



Trtle Given to Evaluation Unit 
. Within LEA or SEA 



Functional words 
included in title 



Smaller ^ 
LEAs 
(n^30) 



Larger 
LEAs 
(n:^25) 



SEAs 
(n=14) 



■ '■ . " '" ~ 

.1 « 

Research and/or evaluation only ' 




13 . 


14 




4 


« 

Research and/or evaluation, plus 
testing and/or assessment 

* . • ^ 




3 


2 • 

Mt 




" 2 


Research and/or evaluation, plus 
planning v 

> 




1 


5 




' 4 


He'search and/or evaluation, plus 

information systems or services 






2 




• 

0 


Other words ipr combinations^ 

>. 




6 


1 


♦ 


• 1 


No response 




7 


1 




3 



1 



Includes such titles as Research and Special Projects, Management Information 
Service^, Program Asses sment<^anch,( Rosejtrch and Develdpment* 
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Table 4 ' 



Title Given to Head of Evaluation Unit " i 



, ■ -v^ — ' — ■ — ■ ■ 

Title • 


Smaller 
LEAs 
Cn-30) 


Larger 
. LEAs 
(nf25) 


SEAs 
(nfl4) 


Director 




■ \ 

12 . 


' '< 
6 


Goordinator 

• 




*• 

. !• 




Supervisor 


4 


1 


0 


Assistant Superintendent | 


P 


• 4 


0 


Other ^ 




.4 


.2 


, '. 








No .response 


9 




4 


^. . . ^ ^ . — . ^ ^ — 

^Inuludes such titles as Assistant Director, Manager, Bureau Chief, Specialist, 
Administrator, Administrative Assistant. 
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. Table 5 



Person Supervising Head of Evatuation Unit 



♦ • 



Supervisor of 
Evaluation Head 



Smaller 
LEAs 
(n=30) 



Larger 
LEAs 
(n=25) 



SEAs 
• Cn=14) 



Superintendent of Schools 
Assistant Superintendent 
Associate Superintendent 



10 



8 



Deputy Superintendent 



A director 



Deputy Commissioner 



0 



0 



0 



Other ^ 



No response 



0 



2 



2 
3 



^Includes such titles as Head of Research and Planning, Senior Managciiieat Officer, 
and Assistant Secretary for Planning. 
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Table 6 


<> 


2 ' * 






Methods Used to^ Determine Program Evaluation Budget in Each Type 


of Agency 








Method 


$mallo^ Larger 

LliAs LliAs Total, LEA* only 
(n - -24) (n - 24) ■ ClL - 52) 


SCAs t 

, , (n " 11] , • , 


Total , 
UiAs HiiU SUAs 
(n - 63) 






w*i"g using using range of V*. 
method method method used « 


\ Actual \, or 
using raniie of %s, 
methQi4 used 


* Acjuai \, or • 
using raiigc of t% 
method^ used 






M SO Mdn 


Value 


^ 

M SD Mdn 



ERIC 



A rouj^hly fixed perccntngc of progroA costs 
it used (give estimate*- ^\)' 



An aflwujtt is determined by the »cope of 
evaluation Jiork jtgivo estimate oTrSagt 



of percentages £i program costs-* 
to 



/ 



"'"^^ il possible , since sufficient 
amount is seldom received (rIvo 
estimate of range of percentages-* 

% to \) 



Other method (specify), Exonplet 
inciudod "ill three of the above/* 
*'no fixed rvlt/' need to consider 
siliry levels of available staff » 



25 



54 



25 



21 



21 



58 



21 



23 



56 



IS 



21 



4.8 



2.8 



4.9 

Lo Hi 
Mdn Mdn 
l.S 

5.4 

Lo Hi L 

Mdn Mdn 



3«2 



9 

J 

46 



36 



27 



Lo 
Mdn 



Hi 
Mdn 

5.0 



21 -4.5 

2.8 

4«8 

Lo HI 
Mdn Mdn 
1.5 ~5 
5.3 



54 



Lo Hi 
Mdn Mdn 

n — 

3.0 



19 



22 



Lo Hi 
Mdn Mdn 

5.1 



Noie, Some respondents indicated using sore thait one method. The number of people itidicmting they used a ptrtlcuUr Mthod was usually slightly latficr 
thin the number who wi^^t on to report the actual I, or range of It, used* k-* ^ * i^un«» w«» usuaiiy eiigniiy larger 



.5^ 





tTl 




< 












c 




ft 












o 
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Amounts IHAs fiudgotod for Evaluation of PodertUy Funded Progrmni* 
as Proportions of Total LUA Budgets Por These Programs, 1973-79 



Federally 
funded 
progiKui 



Smaller LEAs 
(51 . 3a) 

n Program Evaluation % program 

haVing budget budget budgeted 

program (thous* $) (thous« $) for eval* 

Lo Mdn Hi U) Mdn Hi Mdn tfj 



Larger LllAs^ 
(n - 25) 

n ' Program Evaluation \ program 

having budget budget budgeted 

program (thous* $) (thous, $) fot eval* 

Lo Mdn Hi U> Mdn Hi Mdn 



1- 



Total LEAs' 
(rt - 55) 



n Program 
haVing budget 
program (thous A $) 

Lo Hi 



Evaluation % program 
budget budgeted 
(thous, $) for eval* 

Lo ^kln Hi . Mdn ' 



ESEA, Title I, for 

diladvantaged 

students 



21 



104 



700 



3,234 



13 



92 



1.5% 



21 



1*078 



4,770 



17 



52,000 



100 



935. 



2% 



42 



104 



2^00 



52,000 



33 



935 



1,5% 



ESBA, Title I, for 
migrant students 



46 



150 



425 



3 . 



1% 



48 



290 



798 



41 



4,5% 



10 



46 



200 



798 



41 



. 3% 



ESEA, Title IV-C, 
.innovatlv«\, 
curricula 



ESEA, Title Vn»- 
bilingual programs 



CSAA, Emergency 
School Aid Act prog' 
for desegrtgutiiig 



F.L, 94-142, 
special" education 
programs 




10 



315 



440 



100 



26 



7% 16 



1.5% n 



2% 12 



600 



43 



7S0 



.2,112 



107 

I 



390 



7,372 



140 5 350 

4 5?0 .8 1.$% 15 1,410 

4,354 31 



9,400 



lit), V 

510 

t 10,254 



17 



18 



37 



66 



150 



231 



299 



4% 



3% 



3% 



.5% 



29 



19 



25 



85 



87 



2,112 



380 



7^372 



140 

19 1^300 



80 



9/400 



24t^ 

• ^ 10,254 



11 



0 

# 16 



51 



100 



150 



299 



•$% 



5% 



ft , # 



lIHVI* ^ doslgrv^tu. thM lowest and, highest values, respoctlvely^ reported Cor each budget item by eaoK LM category, Por each LEA having ^ * 

pnxticuliir. federal program? the percentage , of program budget alloceted for ovaluaiipn was computed, Entered-in this table are the mediant of these - S 
percentiyjes budgeted for evaluation* ^ ' -.i,. 
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Table 8 

Percentages ^of Evaluation Funds Spent on f^ach Evaluation ^has.e ' 

for •'T>'pical*', Evaluation^ / . 



• Question: 
Estimate the v of total 
program evalua.tion niouies 
spent on each phase ^of " . 
progruirt evaluation for a ,. 
."typical" program • 
evaluation: 



Smaller 
.LEAs 



Larger 
LEAs 



V 0 ; i v'. 



Total, 
LEAs only 



(n. = 26) (n = 23) , (n = 49) • 
M SD. Mdn ♦"•fr SD Mdu M SD Mdn 



SEAs 



(n -^ 8) 
M SD Mdn 



Jotal,. 
• ' LJEAs ■ 
'and SEAs 
. (nn57) 
M SD Mdn 



Program pjlanning/needs 
assessment 



18 



14 



16 



15 



16 



10. 



12 



IQ 



24 



13 



21- 



rl8 . 



12 



^' .15 



Process/imp l.ementat ion 27 - 

data • . 23 



20 



.32 



17- 



31 



30 



20 



27 



.25 



16 



23 



29 



20 V : 

27 



Product/outcome data 



43 



25 



.36 



43 



19 



45. 



43 



22 



40 



33 



21 



28 



41 



22 



40 



Dissemination/use of ' 12 
data 10 



10- 



10 



;5 



Id 



11 



8 



10 



15 



13 



12 



IP 



10 



.*valuati^on Direct(w:s' Judgments Concerning , 
♦ the Impor^nce bf Evaluation Reports , 



Question: Smaller " Larger, Total, , ' SEA.s Total 

/.you juilge your ; ^ LjliAs v V ; LLAs . LEAs only ^ LEAs and SEAs 

^"unit's evaluation • (n'= 30) • •"'(n = 25) (n « 55), (n =«|?4) ( n = 69) 

reports are. an i •/ ■ :\ ; '. % - . % 

» important factor-- • Yes" ..^o . Ye6 I^o -Yes No Yes ^N'o 'Yes No 



— *r^r — ^-r 




Ifi^Vdet'crmining leVel 

f\iSdif( 
^y6*t«?f 



4« 



1. 



^^oif^jpy^alf program- 45 ; . , ^8. • • 45 29 -^l^.. 

* fVljl(cl|ffg in subsequent SO ". ' 44 47 - "'jl .>S7- >- 49 



. In the way the pro^rapi ■ - 

i$' oj^atcd in .subse- ' 87 68 ■ ' ' ^178- " " i '36 70 

^ij^frt* years? ' % "16 .11;' 29 15 



Note., The response ^options were "^esV and "No." The 'difference Yqr each group and each 
.questlQn btftweei^ lOO^j and thfe sum of the Yes and No response. %s, represents the* % not 
, answ^Tihg.' ♦* .v ' ■ " . ... ' ' ■ • 



0 



Table 10 



Evaluation Director's* Policy? Recommendttioi^ 
Concerning Allo;:ation Pro^Vam Resources lo Evaluatio^i ^ 



* * '■ A- ■ 



Qdestipn: 
Wf\at position do 
yo4Ui»reconunend this 
Task Force support? 



' Smaller 
• LEAs . 
( n - 29); 



Support a fioced % of 
program costs for ^ 
progr^ evaluation 27,6 

♦ 

•Stt[»p«rt a minimum'^ 
of program costs 
' for program *valuat Lon 17. 2 



LEA3 
• (n; = 22) 

■% 



9.1 



.18.2 



Total , 
LliAs only 
(n = 51.) 
% 



19.6 



-SEAS, 

(n = 13) 
% 




Totals 

■LEAs an(I SLAs 
i ' (n = 6.4) 




17.6 



15.4 



17.2 



^dopt a sliding scale v. I 

with lower^percentnges 
^,as program costs 
•exceed certain lesvels 51.7 



S9.1 



54.9 



53.8 



54.7 



' Other (Spec 1 f y ) \\ 10.3 



18.2 



13.7 



3a. 8 



17.2 



Note. Some respondents made more than one recommendation. 
_____ ^ 
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Table 11. 

Lvaliuition Directors' Recoimnendcd Percentages 
of Program Costs To Be Budgeted For Evaluation 



* 'T: ' " Question; . 

What ?d of program . . 
' costs should be 
budgeted for prograjri 
ieyaiuation?- - . .^tf ' i 



Smaller 
LLAs 



Lar^'.er 
LEAs 



Total, 

Ulhs 
only 



(n - 25) (n =2;) (n*= 46) 



SliAs 



Cn - 11). 



Total, 
LLAs and 
SliAs 

(n = 5.7) 



JjOwer end of % range 



to w e S t % . YU Jl uo 


. 0 


.1 ■ * 


• 0 


3 


9 0 




Mean' low ' V yalUe 


4.5 


3.0 ' 




4.9 


4.0 ■ 




Median, low % value 


4.3 


2.3 


3.1 


4.7 


3.8 




Standard deviation 
of low % values 


3.9 


. 1'.9 


/ 

3.2 ' 


1.9 


3.0 . 




ilVghor end of % ranfee 




1 


♦ 








Highest % value 


' 20 


15 


20 • . 


12 


. 20 




. Mean- high % value 


■ 8.1 . 


. 7.3 


• 

7.7 . . 

• 


9.2 " 


8.0 


f 


Mcdlttn high %"value 


7.3 


8.0 




9.7 


9.6 


• 


Standard deviation 
• of high S values 


■''5.4 


3,6 


• 4.6 


2.0 


4.3 
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